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Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  0MB  control  number. 
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TUTORIAL 


PRICE  ENUMERATION  AND 
PROBABIIISTIC  EVALUATION 
IN  SYSTEM  ACQUISITION 

Maj  Kevin  Gaudette,  U5AF  and  Kevin  Sweeney 


A  number  of  environmental  uncertainties  in  system  acquisition  can  lead  to 
situations  where  exact  requirements  are  difficult  to  estimate.  The  uncertainties 
are  often  disregarded  in  favor  of  a  simple  point  estimate. This  point  estimate,  or 
Best  Estimated  Quantity  (BEQ),  is  then  used  in  the  Request  for  Proposals 
(REP)  and  subsequent  source  selection,  with  the  evaluation  conducted  at  the 
BEQ  price  using  deterministic  techniques.  An  alternative  approach  is  presented 
for  BEQ  and  deterministic  price  evaluation. The  approach  involves  the  solicitation 
of  a  range  of  bid  prices  for  all  potential  quantities  in  lieu  of  a  BEQ,  with  a 
complementary  probabilistic  analysis  technique  in  lieu  of  a  deterministic  price 
evaluation.  The  probabilistic  approach  is  then  used  to  evaluate  the  range  of  bid 
prices  and  make  an  award  decision. The  methodology  is  presented  in  the  context 
of  an  actual  case  study  in  which  it  was  implemented  in  1996. 


Many  acquisition  programs  face  a 
challenging  and  ever-changing 
environment.  Uncertain,  dy¬ 
namic  requirements  make  program  deci¬ 
sions  and  cost  estimates  extremely  diffi¬ 
cult.  This  difficulty  is  magnified  at  the 
time  of  a  source  selection,  when  the  re¬ 
quirements  must  he  frozen  for  communi¬ 
cation  to  potential  bidders.  In  a  typical 


source  selection,  proposals  are  evaluated 
at  the  quantity  that  is  certain  at  the  time 
of  the  Request  for  Proposal  (RFP),  regard¬ 
less  of  how  certain  that  quantity  is. 

This  quantity,  often  called  the  Best 
Estimated  Quantity  (BEQ),  is  therefore 
simply  a  crude  point  estimate  made  at  a 
distinct  point  in  time.  When  quantities  are 
certain  and  static,  as  is  sometimes  the  case. 


DISCLAIMER 

The  views  expressed  in  this  article  are  those  of  the  authors  and  do  not  reflect  the  official 
policy  or  position  of  the  United  States  Air  Force,  Department  of  Defense,  or  the  U.S. 
government. 
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the  BEQ  is  a  perfectly  acceptable  meth¬ 
odology.  When  program  characteristics 
make  requirements  uncertain  and  dy¬ 
namic,  however,  the  use  of  a  BEQ  and 
deterministic  evaluation  techniques  drives 
cost  risk  to  unnecessary  levels.  In  fact,  the 
only  thing  certain  about  a  forecast  made 
in  an  uncertain  environment  is  that  it  will 
be  wrong.  This  is  an  important  concern, 
because  inaccurate  quantity  estimates 
result  in  one  of  only  two  possible  out¬ 
comes,  illustrated  by  the  “bath  tub”  curve 
in  Eigure  1 . 

In  the  first  outcome,  the  quantity  re¬ 
quired  is  lower  than  the  estimate  used  and 
evaluated  in  the  source  selection.  Three 
things  can  happen  in  this  case.  If  the  lesser 
quantity  was  not  priced  in  the  contract, 
then  the  buyer  is  forced  to  renegotiate  and 
will  likely  pay  a  significantly  higher  price. 
If  the  lesser  quantity  was  priced  in  the  con¬ 
tract,  it  is  still  likely  that  the  unit  price  will 


be  significantly  higher.  This  stands  to  rea¬ 
son,  given  that  the  bidders  will  seek  to  re¬ 
duce  their  own  risk  and  that  price  is  nor¬ 
mally  evaluated  only  at  the  BEQ.  Einally, 
the  buyer  may  be  forced  to  buy  excess 
items  to  satisfy  the  terms  of  the  contract 
and  avoid  default  penalties. 

The  second  possible  outcome  is  that  the 
quantity  needed  increases.  As  was  the  case 
with  the  overestimated  quantity  described 
above,  if  the  higher  quantity  is  priced  in 
the  contract,  it  is  likely  to  be  a  higher  unit 
price.  If  not  priced,  the  buyer  must  either 
conduct  a  new  source  selection  for  the 
additional  units  or  must  renegotiate  the 
existing  contract.  Either  option  will  result 
in  a  higher  price  than  if  the  higher  quan¬ 
tity  had  been  considered  in  the  original 
procurement. 

The  increased  cost  risk  described  above 
is  often  reduced,  albeit  to  a  limited  extent, 
by  the  use  of  such  techniques  as  Variable 


Figure  1.  Notional  Example  of  the  Relationship  Between  Unit  Price 
and  Actual  Quantity  Purchased  for  BEQ  and  Prohahilistic  Approaches 
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Indefinite  Quantity  (VIQ)  contract  options 
and  limited  liability  termination  clauses. 
Unfortunately,  these  techniques  are  partial 
solutions.  The  latter  is  a  “safety  nef’ 
designed  to  limit  the  government’s  liabil¬ 
ity,  while  the  bid  prices  in  the  former  are 
usually  not  incorporated  effectively  into 
the  proposal  evaluation.  These  problems 
can  be  more  rigorously  addressed  using  a 
simple  probabilistic  approach.  This  paper 
describes  the  use  of  one  such  approach  in 
system  acquisition  and  presents  a  case 
study  in  which  it  was  developed  and 
applied.  The  method,  which  abandons  the 
point  estimate  methodology  altogether, 
was  successfully  used  in  a  $200  million 
source  selection  for  the  Joint  Tactical 
Information  Distribution  System  (JTIDS) 
in  1996. 

For  simplicity,  the  approach  is  hereafter 
referred  to  as  PE^.  The  acronym,  although 
admittedly  contrived  for  convenience, 
nevertheless  captures  both  primary  com¬ 
ponents  of  the  methodology  described  in 
this  paper.  The  first  “PE”  is  the  request 
for  an  all-inclusive  set  of  proposal  prices 
covering  the  entire  range  of  possibilities, 
called  price  enumeration.  The  second  is 
the  use  of  probabilistic  techniques  to 
evaluate  that  set  of  proposal  prices,  called 
probabilistic  evaluation.  When  used  to¬ 
gether  (PE^),  the  efficacy  of  these  two 
techniques  is  maximized  for  reasons  that 
will  become  clear. 

The  remainder  of  the  paper  is  organized 
into  four  major  sections:  Case  Study, 
Methodology,  Discussion,  and  Results. 
The  Case  Study  section  provides  an  over¬ 
view  of  the  program  in  which  the  meth¬ 
odology  was  implemented  and  describes 
some  of  the  unique  program  characteris¬ 
tics  that  made  it  necessary.  The  Method¬ 
ology  section  describes  the  method  in 


detail  using  supporting  calculations, 
examples  from  the  case  study,  and  a  brief 
discussion  of  its  advantages.  The  Discus¬ 
sion  section  olfers  a  discussion  of  the  gen¬ 
eral  program  conditions  that  make  proba¬ 
bilistic  methods  attractive.  Strengths  and 
potential  pitfalls  are  included  to  aid 
acquisition  professionals  in  implementa¬ 
tion  efforts,  finally,  the  Results  section 
reports  the  outcome  of  the  case  study 
source  selection. 

Case  Study 


Program  Background 

PE^  was  implemented  during  a  1996 
source  selection  for  JTIDS,  an  Acquisi¬ 
tion  Category  ID  joint  program.  The  pri¬ 
mary  function  of  JTIDS  is  to  distribute 
tactical  information  in  digital  form.  Spe¬ 
cifically,  JTIDS  terminals  provide  jam- 
resistant  digital  commu¬ 
nication  of  data  and 
voice  for  command  and 
control,  navigation,  rela¬ 
tive  positioning,  and 
identification.  JTIDS 
technology  also  locates 
and  identifies  subscrib¬ 
ers  with  respect  to  other 
users  and  is  capable  of 
transmission  rates  far 
above  those  of  most  ex¬ 
isting  communication 
systems.  In  effect,  the 
system  gives  airmen,  sailors,  and  soldiers 
areal-time  “God’s-eye  view”  of  the  battle¬ 
field  (Scott,  1996).  JTIDS  platforms  range 
from  Marine  ground-based  trucks,  to  Navy 
ships,  to  Air  force  fighters.  Airborne 
Warning  and  Control  Systems  (AWACS), 
and  Joint  Surveillance  Target  Attack  Radar 


"The  acronym 
[PE‘],  although 
admittedly 
contrived  for 
convenience, 
nevertheiess 
captures  hath 
primary  cempe 
nents  of  the 
methedeiegy 
described  in 
this  paper." 
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Systems  (JSTARS).  The  1996  source 
selection  was  held  to  award  the  final 
production  contract  of  JTIDS  Class  2/2H 
terminals.  Two  contractors  had  been  in¬ 
volved  in  the  research  and  development, 
low-rate  initial  production,  and  first  full- 
rate  production  contracts,  and  they  were 
the  only  companies  realistically  capable 
of  performance. 

Program  Characteristics 

The  JTIDS  program  has  many  charac¬ 
teristics  that  illuminate  the  advantages  of 
using  a  probabilistic  methodology.  All 
four  U.S.  military  services  had  expressed 
requirements  for  10  different  variants  of 
the  terminals  and  had  personnel  working 
in  the  Joint  Program  Office  (JPO)  at 
Hanscom  Air  Force 
Base.  In  addition,  there 
were  several  NATO  cus¬ 
tomers,  one  of  which 
was  physically  repre¬ 
sented  on  the  JPO  staff. 
The  diverse  customer 
base,  each  with  its  own 
independent  line  of 
funding,  made  require¬ 
ments  forecasting  ex¬ 
tremely  difficult.  Projections  would  often 
change  on  a  daily  basis,  depending  on  the 
funding  posture  of  each  service  and  NATO 
customer. 

Since  JTIDS  is  a  subsystem  installed 
on  various  land,  sea,  and  airborne  plat¬ 
forms,  the  requirements  were  also  subject 
to  fluctuations  in  projected  platform 
end-strengths.  In  addition,  the  final  full- 
rate  production  (FRP)  contract  spanned 
five  years,  making  forecasts  even  more 
difficult.  Firm  requirements  were  a  rela¬ 
tively  simple  matter  for  the  first  year,  but 
for  the  four  subsequent  years  the  services 


"The  JTIDS  pro¬ 
gram  has  many 
characteristics 
that  iiiuminate 
the  advantages 
of  using  a 
prehahiiistic 
methedeiegy." 


were  reluctant  to  commit  due  to  budget 
uncertainties  in  the  federal  Planning,  Pro¬ 
gramming,  and  Budgeting  System 
(PPBS). 

In  combination,  the  diversity  of  the 
customer  base,  budget  uncertainties,  and 
the  time  span  of  the  contract  made  the 
quantities  nearly  impossible  to  forecast. 
This  highly  uncertain  and  dynamic  envi¬ 
ronment  rendered  the  traditional  BEQ 
approach  to  estimating  quantities  for  a 
long-term  contract  inadequate.  A  proba¬ 
bilistic  approach  was  therefore  used  to 
reduce  program  cost  risk,  while  ensuring 
a  reasonable  price  for  all  JTIDS  users. 


Methodology 


The  Discrete  Probability  Distributioh 

As  discussed  previously,  PE^  actually 
consists  of  two  techniques  used  in  tandem 
—  price  enumeration  and  probabilistic 
evaluation  of  the  complete  set  of  prices. 
The  two  are  technically  independent,  but 
their  symbiotic  relationship  dictates  that 
they  be  used  together  for  maximum  ef¬ 
fectiveness.  A  complete  range  of  prices 
can  certainly  be  solicited  without  using 
probabilistic  techniques  to  evaluate  them, 
although  it  makes  little  sense  to  do  so  in 
practice.  Eikewise,  any  probabilistic  tech¬ 
nique  could  be  employed  in  an  evaluation 
even  if  a  limited  number  of  bid  prices  are 
solicited  in  the  REP,  but  this  limits  the 
efficacy  of  the  technique  since  the  full 
range  of  potential  prices  will  not  be 
explicitly  included  in  the  contract. 

It  should  be  noted  that  the  evaluation 
technique  described  in  this  section, 
although  chosen  for  several  reasons  that 
will  be  discussed,  is  only  one  of  many 
techniques  that  are  available.  Alternative 
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techniques  like  Monte  Carlo  simulation, 
stochastic  math  programming,  and  simple 
sensitivity  analysis,  used  individually  or 
in  comhination,  can  all  perform  equally 
well.  They  can  also  he  used  to  enhance 
PEI 

The  prohahilistic  evaluation  used  in  the 
JTIDS  program  involves  a  technique  em¬ 
ploying  a  discrete  prohahility  distribution 
and  expectation  values.  The  primary  rea¬ 
son  that  this  technique  was  chosen  over 
competing  prohahilistic  techniques  for  the 
JTIDS  source  selection  is  simplicity.  Tra¬ 
ditional  methods  like  those  mentioned 
above  are  excellent  tools  for  analysis  of  a 
set  of  possible  outcomes,  but  they  can  be 
unnecessarily  complex  for  the  problem  at 
hand.  Additionally,  the  results  can  be  dif¬ 
ficult  to  interpret  and  explain  to  decision 
makers.  Since  the  solicitation  of  prices  for 
all  possible  quantities  is  a  straightforward 
concept,  it  is  omitted  from  the  method¬ 
ological  discussion  for  brevity.  Imple¬ 
mentation  issues  associated  with  bid  price 
enumeration  are  discussed  in  the  Discus¬ 
sion  section,  but  for  now  the  focus  is  di¬ 
rected  toward  the  probabilistic  evaluation 
technique. 

In  the  context  of  a  source  selection  for 
uncertain  quantities,  a  discrete  probabil¬ 
ity  distribution  is  composed  of  a  finite  set 
N  of  quantities  q,  for  each  of  M  line  items 
—  each  with  an  associated  probability  of 
occurrence  p(q).  The  expected  contract 
cost  E(C)  is  simply  the  sum  of  the  prod¬ 
ucts  of  all  possible  outcomes  and  their  as¬ 
sociated  probabilities  of  occurrence, 
summed  over  all  line  items  in  the  contract 
(Ross,  2000).  The  equation  illustrates  this 
mathematically. 

M  N, 

E(,C)  =  Y^{q„c,.p(q,p) 

i=l  j=l 


Where 

q.,  -  j"’  possible  value  of  q  for  line 

item  i 

c..  -  bid  unit  price  (cost)  when 

buying  j  units  of  line  item  / 

p( ^  j  =  probability  of  buying  j  units 
of  line  item  i 

M  -  number  of  different  line  items 
on  contract 

N.  -  number  of  possible  quantities 
for  line  item  / 

E(C)  -  expected  cost  of  the  contract 

For  each  item,  it  is  improtant  to  note 
that  the  sum  of  all  probabilities  must 
equal  one. 

=  l  Vi 

./=1 

Although  the  equation  represents  one 
of  the  most  basic  concepts  in  probability 
theory,  its  use  is  predominantly  absent 
from  military  acquisition.  This  simple 
equation,  however,  can  significantly  re¬ 
duce  the  cost  risk  in  a  program  if  used 
properly.  By  explicitly  incorporating 
prices  for  all  potential  quantities  in  the 
price  evaluation,  direct  incentive  is  pro¬ 
vided  to  bidders  to  offer  reasonable  prices 
across  the  entire  range  of  quantities.  In  the 
JTIDS  source  selection,  the  equation  was 
incorporated  into  a  spreadsheet  that  was 
distributed  on  a  floppy  disk  with  the 
Request  for  Proposal  (REP)  (See  Figure 
2).  The  formulas  and  probabilities,  there¬ 
fore,  were  fully  visible  to  the  bidders.  Al¬ 
though  hundreds  of  bid  prices  were 
required,  the  bidders  could  easily  fill  in 
the  blanks  and  conduct  internal  “what  if’ 
analyses  to  arrive  at  their  desired  bottom 
lines. 
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Qty 

Weight 

% 

Total 

Price 

Avg  Unit 

Price  (AUP) 

Weighted 

AUP 

0 

45.00% 

$200,000 

$0 

$0 

1 

25.00% 

$200,000 

$200,000 

$50,000 

2 

20.00% 

$350,000 

$175,000 

$35,000 

3 

10.00% 

$450,000 

$150,000 

$15,000 

Figure  2.  Sample  Spreadsheet  Entry  for  JTIDS  Class  2  Terminals, 
Configuration  1  (F- 1 5/MAOC),  Fiscal  Year  1997 


In  Figure  2,  for  example,  the  hypotheti¬ 
cal  bids  show  a  nonincreasing  unit  price 
for  increasing  quantities.  This  example 
illustrates  the  quantity  discount  one  would 
expect.  Since  the  BEQ  (in  hold  border)  is 
zero  units,  the  F-15  terminal  would  likely 
not  have  been  included  in  a  traditional 
BEQ-based  solicitation.  Its  inclusion  here 
avoids  costly  changes  later  should  the  user 
decide  to  exercise  an  option  to  buy  termi¬ 
nals.  This  example  is  purposely  chosen 
here,  since  an  option  was  in  fact  later 
exercised  for  F-15  aircraft  at  Mountain 
Home  Air  Force  Base  participating  in  a 
Theater  Missile  Defense  (TMD)  Advanced 
Technology  Concept  Demonstration 
(ATCD)  (Scott,  1996). 

Advantages 

A  probabilistic  approach  offers  four  pri¬ 
mary  advantages  over  the  traditional  BEQ 
method.  First,  the  evaluation  price  that 
ultimately  drives  the  cost  evaluation  is  a 
probabilistically-weighted  average  of  all 
possible  quantities.  This  leads  to  a  more 
defendable  forecast  of  the  expected  cost 
of  the  contract  than  does  a  simple  point 
estimate  (Anderson  &  Cherwonik,  1997). 
It  also  offers  incentive  to  bid  fairly  for  all 


quantities,  since  each  will  ultimately 
impact  the  price  evaluation. 

Second,  the  bidders  explicitly  submit 
bids  for  each  possible  quantity.  This  has 
the  obvious  benefit  of  eliminating  nearly 
all  cost  risk  associated  with  quantity  un¬ 
certainty,  since  the  unit  prices  for  all  pos¬ 
sible  quantities  are  contained  in  the  con¬ 
tract.  Floyd  (2000)  was  a  proponent  of  this 
approach  whenever  possible  contract 
changes  threaten  future  prices. 

Third,  it  allows  program  offices  the 
ability  to  proceed  with  a  source  selection, 
if  it  makes  sense  to  do  so,  even  if  the  final 
funded  quantities  are  still  uncertain 
(Floyd,  2000).  In  the  case  of  JTIDS,  if 
the  JPO  had  waited  until  quantities  were 
certain  to  conduct  the  source  selection,  it 
would  have  been  delayed  indefinitely. 

Finally,  any  attempt  to  take  advantage 
of  the  system  using  complex  bidding 
strategies  is  immediately  evident  to  the 
source  selection  evaluation  team.  Should 
some  of  the  bid  prices  be  unreasonable,  it 
is  a  simple  matter  to  translate  that  unrea¬ 
sonableness  into  technical  risk.  This  ad¬ 
vantage  is  consistent  with  the  recent  push 
toward  Best  Value  source  selections,  in 
which  awards  to  a  bidder  other  than  the 
one  with  the  lowest  cost  are  often  deemed 
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advantageous  to  the  government.  Any 
proposed  priees  that  are  considered 
unreasonable  can  he  directly  and  quan¬ 
titatively  penalized  in  the  technical  evalu¬ 
ation  (O’Connor,  Paris,  &  Lovelace, 
1997).  This  alleviates  the  problem  of 
subjective  technical  evaluations  that  fail 
to  stand  up  to  protests  by  the  losing  bid¬ 
ders  —  a  perennial  problem  in  source 
selections  (Lloyd,  2000;  Raymond,  1999). 


Discussion 


Certain  conditions  tend  to  favor  the  use 
of  PE^  in  source  selections.  As  alluded  to 
throughout  this  paper,  the  method  has 
many  advantages  over  traditional  point 
estimate  evaluation  techniques.  Unfortu¬ 
nately,  there  are  also  several  pitfalls  in¬ 
herent  in  the  methodology  that  must  be 
carefully  avoided.  Program  managers  and 
cost  analysts  need  to  recognize  the  condi¬ 
tions  under  which  the  use  of  PE^  is  both 
appropriate  and  preferred,  and  also  must 
have  a  firm  grasp  of  the  strengths  and 
potential  pitfalls.  The  conditions  favoring 
application  of  PE^  are  discussed  first. 

Conditions  Favoring  PE^ 

Although  the  list  of  four  conditions  dis¬ 
cussed  here  is  certainly  not  intended  to 
be  all-inclusive,  it  contains  four  of  the 
most  important  conditions  in  terms  of  any 
probabilistic  methodology.  It  is  not  neces¬ 
sary  that  all  conditions  be  met  for  PE^  to 
be  used,  but  obviously  the  number  met  and 
the  degrees  to  which  they  are  met  are  fac¬ 
tors  that  must  be  considered.  The  four  con¬ 
ditions  are  quantity  uncertainty,  current 
and  accurate  cost  data,  large  quantities, 
and  high  unit  cost.  The  first  two  can  be 
distinguished  from  the  latter  two  as  pre¬ 


requisites,  for  reasons  that  will  become 
apparent. 

Quantity  Uncertainty  -  Quantity  un¬ 
certainty  is  an  obvious  prerequisite  con¬ 
dition  for  the  use  of  probabilistic  tech¬ 
niques,  since  there  is  no  need  for  these 
techniques  if  the  quantity  is  known.  The 
degree  of  uncertainty,  however,  must  be 
evaluated  to  determine  whether  it  is  high 
enough  to  warrant  application  of  PE^.  In 
the  JTIDS  source  selection,  the  degree  of 
uncertainty  was  clear.  The  firm  require¬ 
ments  totaled  just  40  terminals,  represent¬ 
ing  the  minimum  buy  in  the  first  contract 
year.  Over  the  course  of  three  years  of  ter¬ 
minal  buy  options,  how¬ 
ever,  the  maximum 
quantity  was  230  termi¬ 
nals.  The  high  degree  of 
uncertainty  and  range  of 
potential  quantities  war¬ 
ranted  the  use  of  proba¬ 
bilistic  techniques.  Un¬ 
certain  procurement  quantities  in  the 
JTIDS  case  resulted  from  a  diverse  cus¬ 
tomer  base,  budget  uncertainties,  multiple 
platforms,  and  multi-year  contracts;  but 
other  programs  may  experience  other 
causal  factors  leading  to  quantity  uncer¬ 
tainty. 

Current  and  Accurate  Cost  Data  - 

The  second  prerequisite  condition  pre¬ 
sented  here  is  that  of  current  and  accurate 
cost  data.  Because  the  proposals  will  con¬ 
tain  a  large  number  of  bid  prices,  each 
must  be  assessed  for  reasonableness.  Any 
bid  prices  that  are  deemed  unreasonable 
can  then  be  translated  to  the  technical 
evaluation  in  the  form  of  risk.  Without 
current  and  accurate  cost  data,  it  is  very 
difficult  in  practice  to  quantify  that  risk 
and  thereby  discourage  bidders  from  taking 
advantage  of  the  system.  In  the  case  of 


"Certain  conditions 
tend  to  favor  the 
use  of  PE‘  in  source 
seiectiens." 
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JTIDS,  the  maturity  of  the  program  meant 
that  the  JPO  had  a  great  deal  of  historical 
cost  data  to  use  in  the  evaluation. 

Large  Quantities  -  A  large  procure¬ 
ment  in  terms  of  quantity  can  also  war¬ 
rant  the  use  of  PE^  or  similar  techniques. 
For  such  contracts,  even  a  small  percent 
deviation  from  the  estimate  can  represent 
a  significant  increase  or  decrease.  Again, 
in  the  case  of  JTIDS  the  range  of  quanti¬ 
ties  for  terminals  was  190,  with  a  firm 
(minimum)  requirement  of  40  and  a  maxi¬ 
mum  of  230.  If  only  10  percent  of  the 
uncertain  quantities  were  subsequently 
purchased,  the  total  huy  would  increase 
hy  19  terminals.  This  represents  nearly  a 
50  percent  increase  in  units  from  the  firm 
BEQ  originally  contemplated  for  the 
source  selection. 

High  Unit  Cost  -  A  related  hut  distinct 
condition  is  the  unit  cost 
of  the  items  in  the  con¬ 
tract.  Just  as  large  quan¬ 
tities  can  he  significant 
even  at  low  unit  costs, 
large  unit  costs  can  he 
significant  even  at  low 
quantities.  JTIDS  termi¬ 
nals  cost  between  $50 
thousand  and  $180  thousand  each,  de¬ 
pending  on  the  type,  so  even  a  deviation 
of  one  terminal  can  translate  to  a  substan¬ 
tial  increase  in  cost.  When  combined  with 
large  quantities,  the  sensitivity  to  small 
deviations  becomes  even  more  significant. 

Strengths  of  PE^ 

PE^  has  many  advantages.  The  four 
discussed  here  are,  by  nature  of  the  un¬ 
derlying  conditions,  the  most  important. 
These  are  risk  reduction,  visibility,  ease 
of  implementation,  and  flexibility. 


"PE‘  has  many 
advantages  ••• 
risk  reduction, 
visibiiity,  ease  of 
impiementatien, 
and  fiexibiiity." 


Risk  Reduction  -  Risk  reduction  is  the 
most  obvious  advantage,  and  in  fact  the 
one  that  drove  the  use  of  probabilistic 
techniques  in  the  JTIDS  program.  In  any 
program  where  quantity  uncertainty  is 
present,  program  managers  run  the  risk  of 
paying  higher  prices  when  quantities 
change  (Eloyd,  2000).  PE^  can  reduce  or 
even  eliminate  that  risk  by  requiring  prices 
for  all  possible  quantities  at  the  time  of 
proposal,  rather  than  adjusting  those 
quantities  later.  The  possibilities  are  ex¬ 
plicitly  enumerated  and  probabilistically 
evaluated  from  the  beginning. 

Visibility  -  Because  the  complete  set 
of  prices  is  required  in  the  proposal,  any 
attempt  to  take  advantage  of  the  system  is 
immediately  visible  to  source  selection 
evaluation  team  members.  Where  under¬ 
lying  bidding  strategies  are  masked  to  the 
team  using  the  BEQ  method,  they  are 
uncovered  using  enumerative  pricing.  This 
makes  the  quantitative  assessment  of  risk 
straightforward  and  defendable,  avoiding 
the  traps  that  a  subjective  technical  evalu¬ 
ation  can  bring  (Eloyd,  2000). 

Ease  of  Implementation  -  PE^  is  rela¬ 
tively  straightforward  to  implement.  Al¬ 
though  its  application  to  source  selection 
is  unique,  the  equation  is  a  mathematically 
simple  concept.  The  evaluation  price  can 
easily  be  described  to  decision  authorities 
as  a  probabilistically-weighted  average, 
and  therefore  should  be  readily  accepted 
by  program  managers,  cost  analysts,  and 
contracting  officers  alike.  As  demon¬ 
strated  in  Figure  2,  the  equation  can  be 
incorporated  into  a  spreadsheet  and  dis¬ 
tributed  to  potential  bidders  electronically 
with  the  REP 

Flexibility  -  Finally,  PE^  is  flexible.  In 
some  programs,  quantity  uncertainty  may 
exist  only  for  a  subset  of  contract  line 
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items.  There  may  also  be  more  uncertainty 
with  some  items  than  with  others.  In 
JTIDS,  the  degree  of  quantity  uncertainty 
varied  both  by  terminal  type  and  by 
customer.  The  F-14  variant,  for  example, 
had  no  firm  or  expected  requirements  and 
a  maximum  of  nine  terminals  over  three 
years.  This  was  due  to  the  fact  that  the 
Navy  was  in  the  process  of  developing  the 
next  generation  Multifunctional  Informa¬ 
tion  Distribution  System  (MIDS),  but 
wanted  to  retain  an  option  of  purchasing 
a  few  JTIDS  terminals  should  the  MIDS 
schedule  be  delayed.  In  contrast,  the  Navy 
had  a  firm  requirement  for  nine  shipboard 
terminals,  with  an  expectation  to  buy  an 
additional  10  in  the  second  year  and  a 
maximum  possible  requirement  of  48.  PE^ 
may  not  have  been  justified  for  the  F-14 
terminals  alone,  but  it  is  flexible  enough 
to  be  used  for  many  uncertainty  levels 
under  the  umbrella  of  the  same  contract. 

Pitfalls 

The  final  consideration  for  implement¬ 
ing  PE^  is  its  potential  challenges  or  pit- 
falls.  There  were  many  challenges  encoun¬ 
tered  in  the  development  and  refinement 
of  the  implementation,  but  three  were  by 
far  the  most  problematic.  Coordination 
and  approval  of  the  methodology,  price 
execution,  and  probability  estimation  are 
discussed  below. 

Coordination  and  Approval  -  Given 
the  preceding  discussion  on  the  ease  of 
implementation,  it  may  seem  somewhat 
contradictory  that  the  coordination  and 
approval  of  the  use  of  PE^  is  listed  as  a 
potential  pitfall.  This  is  not  a  reflection  of 
the  complexity  of  the  methodology,  but  is 
instead  a  reflection  of  the  entrenched  para¬ 
digms  of  point  estimates  and  BEQ.  Since 
the  technique  has  not  historically  been 


used  in  system  acquisition,  there  is  a  natu¬ 
ral  organizational  resistance  that  must  be 
overcome.  Although  this  was  initially  a 
secondary  concern  in  the  JTIDS  source 
selection,  the  proper  vetting  of  the  con¬ 
cept  required  a  great  deal  of  time  and  effort 
in  practice.  Only  when  the  source  selec¬ 
tion  and  subsequent  protest  were  complete 
were  the  cost  analysis  and  program  man¬ 
agement  directorates  convinced  of  its 
efficacy. 

Price  Execution  -  A  second  pitfall  to 
avoid  can  be  termed  “price  execution,” 
since  it  involves  the  actual  execution  of 
the  contract  and  its  associated  prices.  For 
each  line  item,  the  contract  will  stipulate 
unit  prices  that  are  a  function  of  the  total 
number  bought  in  a  given  fiscal  year.  The 
total  number  to  be  bought  should  there¬ 
fore  be  known  at  the  time  of  the  first  pur¬ 
chase  in  order  to  take  advantage  of  quan¬ 
tity  discounts.  In  other  words,  the  buyer 
must  hypothetically  or¬ 
der  all  quantities  at  once. 

For  JTIDS,  since 
multiple  customers  were 
involved  and  quantities 
often  changed  continu¬ 
ously  throughout  the 
year,  ordering  a  definite 
number  once  each  year 
was  not  technically  fea¬ 
sible.  To  address  this 
problem,  a  provision  was  inserted  to  re¬ 
quire  the  government  to  commit  by  a  cer¬ 
tain  cutoff  date  in  each  year  of  the  con¬ 
tract.  The  date  needed  to  be  early  enough 
to  determine  the  price  level  for  that  year’s 
buy,  but  late  enough  to  ensure  that  the 
budget  had  been  resolved  before  any  or¬ 
ders  were  placed.  With  that  in  mind,  an 
ordering  deadline  in  January  was  deemed 
reasonable  for  JTIDS,  given  that  the 


"Ceerdinalieii 
and  approval  of 
the  melhedelegy, 
price  execution, 
and  probabiiily 
estimation... 
were  tbe  most 
probiematic." 
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federal  government’s  fiscal  year  begins  on 
1  October. 

This  solution  is  relatively  straightfor¬ 
ward  in  practice.  All  customers  commit 
to  a  quantity  by  the  deadline  and  obligate 
funds  to  cover  that  quantity.  Beyond  the 
ordering  deadline,  the  buyer  pays  the  unit 
price  associated  with  the  original  quan¬ 
tity.  For  example,  if  the  buyer  can  com¬ 
mit  to  buying  10  units  by  the  deadline,  the 
price  at  10  units  is  used  even  if  additional 
units  are  subsequently  purchased  in  that 
fiscal  year.  Interestingly,  the  winning 
JTIDS  bidder  offered  a  flat  price  for  each 
terminal,  regardless  of  the  quantity  pur¬ 
chased,  so  the  contractual  deadline  was 
not  necessary  in  the  end.  In  fact,  the  meth¬ 
odology  actually  encourages  this  type  of 
bidding  strategy,  as  illustrated  in  Figure 
1. 

Probability  Estimation  -  The  final  pit- 
fall  discussed  here  is  that  of  probability 


estimation.  Although  the  use  of  discrete 
probabilities  in  a  stochastic  process  cer¬ 
tainly  has  an  advantage  over  a  single  point 
estimate,  the  realistic  a  priori  estimate  of 
the  probabilities  can  be  a  challenge. 
Unfortunately,  this  process  is  highly  sub¬ 
jective.  In  general,  however,  the  buyer  will 
have  several  subsets  of  requirements  that 
can  help  with  the  estimation  process. 
Market  research  can  help  to  define  these 
subsets. 

One  set  consists  of  firm  requirements, 
either  accompanied  by  obligated  funds  or 
confirmed  in  writing  by  high-level  deci¬ 
sion  makers  in  all  customer  organizations. 
At  the  opposite  end  of  the  spectrum  are 
the  “wish  list”  requirements,  in  the  out 
years  of  the  contract,  that  have  not  been 
included  in  future  budget  estimates  but 
that  the  customer  organizations  are  work¬ 
ing  to  include.  Between  these  two  extremes 


Qty 

Weight 

% 

Total 

Price 

Avg  Unit 
Price  (AUP) 

Weighted 

AUP 

8 

6.25% 

$0 

$0 

9 

6.25% 

$0 

$0 

10 

7.50% 

$0 

$0 

11 

7.50% 

$0 

$0 

12 

45.00% 

$0 

$0 

13 

7.50% 

$0 

$0 

14 

7.50% 

$0 

$0 

15 

6.25% 

$0 

$0 

16 

6.25% 

$0 

$0 

Figure  3.  Probability  Dislribulien  for  E-3  AWACS  Terminals, 

Fiscal  Year  1997 
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there  may  be  several  sets  of  requirements 
that  have  varying  degrees  of  certainty. 

By  stratifying  the  quantities  in  this  way, 
realistic  probabilities  can  be  applied  rela¬ 
tively  easily.  In  addition,  the  use  of 
quantity  zero  (0),  when  market  research 
supports  that  possibility,  should  be  con¬ 
sidered  with  an  appropriate  probability 
applied.  Although  somewhat  counter¬ 
intuitive,  this  option  maintains  a  total 
probability  of  one,  and  therefore  the  math¬ 
ematical  integrity  of  the  method. 

In  the  JTIDS  case,  the  highest  prob¬ 
abilities  were  assigned  to  the  quantities  in 
the  first  year  of  the  contract  for  which 
funds  had  been  obligated.  Other  quanti¬ 
ties  in  the  first  year  were  assigned  lower 
probabilities  as  appropriate.  For  the  sub¬ 
sequent  two  years  of  terminal  buys,  the 
probabilities  were  more  evenly  distributed 
across  the  range  of  quantities  due  to  the 
increased  uncertainty  in  those  years.  Fig¬ 
ures  3  and  4  offer  examples  of  the  two 
cases  cited  above,  respectively,  while  Fig¬ 
ure  4  also  illustrates  the  inclusion  of  a 
quantity  of  zero. 

A  final  word  of  caution  is  warranted 
with  regard  to  the  determination  of  prob¬ 
abilities.  Because  the  process  is  somewhat 
subjective,  the  probabilities  are  susceptible 


to  scrutiny  by  both  decision-makers  and 
bidders.  Decision-makers  will  likely 
question  their  validity  during  the  source 
selection  process,  and  losing  bidders  will 
likewise  focus  any  protest  efforts  on  such 
subjective  decisions.  A  sensitivity  analy¬ 
sis  should  therefore  be  considered  as  a 
means  of  defending  the  probabilities.  This 
analysis  can  be  as  simple  as  a  recalcula¬ 
tion  of  the  evaluation  price  using  several 
sets  of  probabilities  that  are  slightly  dif¬ 
ferent  from  the  primary  set,  or  as  com¬ 
plex  as  a  Monte  Carlo  simulation.  In  any 
event,  the  probabilities  can  be  more 
readily  defended  if  these  techniques  are 
used  throughout  the  source  selection 
process,  rather  than  waiting  until  the 
questions  are  asked. 

Results 


Despite  the  challenges  faced  by  the 
JTIDS  JPO  in  refining  and  implementing 
PE^  in  its  source  selection,  it  worked  as 
designed.  The  details  of  the  evaluation 
methodology  were  clearly  communicated 
in  the  RFP,  so  that  each  bidder  had  a  com¬ 
plete  and  common  understanding  of  the 
process  prior  to  submitting  their  bids. 


Qty 

Weight 

% 

Total 

Price 

Avg  Unit 

Price  (AUP) 

Weighted 

AUP 

0 

30.00% 

$0 

$0 

1 

23.33% 

$0 

$0 

2 

23.33% 

$0 

$0 

3 

23.33% 

$0 

$0 

Figure  4.  Probability  Dislribulien  for  F- 1 4  Terminals, 
Fiscal  Year  1999 
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Examples  were  articulated  clearly  in  the 
REP  and  questions  were  answered  for 
clarification  purposes  wherever  necessary. 
To  ease  the  process  for  both  the  evalua¬ 
tion  team  and  the  bidders,  a  spreadsheet 
accompanied  the  REP  that  contained  all 
quantities,  their  probabilities,  and  the 
formulas  for  calculating  the  total  cost. 

When  the  proposals  were  received,  one 
bidder  submitted  a  proposal  that  attempted 
to  take  advantage  of  the  system.  The  over¬ 
all  evaluation  price  E(C)  had  been  artifi¬ 
cially  lowered  by  the  insertion  of  unreal¬ 
istically  low  prices  for  all  quantities  with 
a  low  probability  of  occurrence.  As  a 
result,  increased  risk  associated  with  the 
low  prices  was  assessed  in  the  technical 
evaluation.  The  contract  was  therefore 
awarded  to  the  competing  bidder,  even 
though  its  evaluation  price  was  higher.  The 
resulting  contract  contained  a  flat  set  of 
prices  for  each  terminal  type,  covering  all 
potential  quantities,  spares,  and  warranty 
repair  throughout  its  five-year  duration. 
The  risks  of  additional  source  selections 
or  renegotiation  of  the  existing  contract, 
and  by  extension  the  risk  of  higher  costs, 
was  effectively  eliminated.  In  fact,  the  PE^ 
methodology  subsequently  withstood  the 
scrutiny  of  the  General  Accounting  Office 
(GAO)  when  the  losing  bidder  submitted 
an  unsuccessful  protest  against  the 
government’s  award  decision. 


Conclusion 


Probabilistic  techniques  have  been  used 
in  a  wide  range  of  operational  settings  for 
decades.  The  success  of  the  1996  JTIDS 
source  selection  in  a  challenging  and 
uncertain  environment  provides  evidence 
of  the  efficacy  of  these  techniques  in 
government  acquisition  programs.  This 
paper  presented  a  methodology  using 
price  enumeration  and  a  probabilistic 
evaluation  of  price  to  address  quantity  and 
cost  risk.  The  methodology  is  ideal  for 
procurements  that  have  one  or  more  of  the 
following  characteristics: 

1.  Quantity  Uncertainty  -  Due  to  a 
diverse  customer  base,  budget  uncer¬ 
tainties,  multiple  platforms,  multi¬ 
year  contracts,  or  other  factors. 

2.  Current  and  Accurate  Cost  Data. 

3.  Earge  Quantities. 

4.  High  Unit  Cost. 

Primary  strengths  of  the  approach  in¬ 
clude  risk  reduction,  visibility  of  bidders’ 
pricing  strategies,  ease  of  implementation, 
and  flexibility.  However,  acquisition  pro¬ 
fessionals  are  cautioned  to  take  time  to  ex¬ 
plain  the  methodology  to  decision-makers 
well  in  advance,  make  appropriate  arrange¬ 
ments  for  executing  the  resulting  contract 
prices,  and  conduct  careful  market  research 
to  estimate  the  probabilities  associated  with 
different  quantities. 
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